Method and system of synchronizing processors to the same computational point

ABSTRACT

A system for synchronizing central processing units (CPU) includes a schedule module that communicates a synchronization point, a first CPU that writes a first memory address to a first register in response to the first CPU reaching the synchronization point, and a second CPU that writes a second memory address to a second register in response to the second CPU reaching the synchronization point. The system further includes a first logical AND module that writes a first value to a third register based on the first and second memory addresses and a second logical AND module that writes a second value to a fourth register based on the first and second memory addresses. The system also includes a scheduler module that selectively generates a processor sync signal based on the first and second value.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/254,930 filed on Apr. 17, 2014. This application claims the benefitand priority of Chinese Application No. 201410008511.8 filed Jan. 8,2014. The entire disclosures of the above applications are incorporatedherein by reference.

FIELD

The present disclosure relates to fault-tolerant failsafe computersystems.

BACKGROUND

This section provides background information related to the presentdisclosure which is not necessarily prior art.

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

An external safety system, such as a railway system, may includefault-tolerant failsafe computer systems configured to implement asafety application. The fault-tolerant failsafe computer systems mayinclude a plurality of hardware components electrically and logicallycoupled in order to implement the safety application. The safetyapplication selectively communicates with safety critical hardware andsoftware. The safety critical hardware and software are configured tocontrol safety relevant functions of the railway system.

For example, a train riding on the railway system includes a brakingsystem. The braking system is configured to implement at least onesafety relevant function, such as a braking function. The braking systemincludes a brake and software configured to actuate the brake. Thesoftware receives instructions to actuate the brake. For example, anoperator of the train may operate a braking system user interface inorder to instruct the software to actuate the brake. Periodically, anerroneous instruction to actuate the brake may be generated by a faultycomponent of the railway system. Accordingly, a fault-tolerant failsafecomputer system configured to validate instructions received by externalsafety systems is desired.

Central processing units (CPUs) in the past had the ability to beinstruction lock stepped. For example, each instruction and data on abackplane or bus would be identical. However, modern CPUs are no longerdeterministic. Multiple CPUs executing the same code may not execute inexactly the same sequence because of multi-threading and cachingtechniques that are used today. Further the frequencies of today'sprocessors far exceed older generations of CPUs. Accordingly, a systemand method for synchronizing CPUs to the same computation point isdesired.

SUMMARY

This section provides a general summary of the disclosure, and is not acomprehensive disclosure of its full scope or all of its features.

A system for synchronizing central processing units (CPU) includes aschedule module that communicates a synchronization point, a first CPUthat writes a first memory address to a first register in response tothe first CPU reaching the synchronization point, and a second CPU thatwrites a second memory address to a second register in response to thesecond CPU reaching the synchronization point. The system furtherincludes a first logical AND module that writes a first value to a thirdregister based on the first and second memory addresses and a secondlogical AND module that writes a second value to a fourth register basedon the first and second memory addresses. The system also includes ascheduler module that selectively generates a processor sync signalbased on the first and second value.

In other features, a method includes writing a first memory address to afirst register, writing a second memory address to a second register,writing a first value to a third register based on the first and secondmemory addresses, writing a second value to a fourth register based onthe first and second memory addresses, and selectively generating aprocessor sync signal based on the first and second value.

Further areas of applicability will become apparent from the descriptionprovided herein. The description and specific examples in this summaryare intended for purposes of illustration only and are not intended tolimit the scope of the present disclosure.

DRAWINGS

The drawings described herein are for illustrative purposes only ofselected embodiments and not all possible implementations, and are notintended to limit the scope of the present disclosure.

FIG. 1 is a functional block diagram of a fault-tolerant failsafecomputer system according to the principles of the present disclosure;

FIG. 2 is a functional block diagram of a fail-safe chassis according tothe principles of the present disclosure;

FIG. 3 is a functional block diagram of a synchronization systemimplementing the synchronization mode according to the principles of thepresent disclosure;

FIG. 4 is a flow diagram illustrating a processor synchronization methodaccording to the principles of the present disclosure; and

FIG. 5 is a flow diagram illustrating an alternative processorsynchronization method according to the principles of the presentdisclosure.

Corresponding reference numerals indicate corresponding parts throughoutthe several views of the drawings.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference tothe accompanying drawings.

Referring now to FIG. 1, a functional block diagram of an exemplaryfault-tolerant failsafe computer system 100 is shown. The system 100 isarranged to interact with safety applications. For example, the system100 is arranged to communicate with safety critical hardware andsoftware associated, by way of non-limiting example, a railway system.The safety critical hardware and software control safety relevantcomponents of the railway system. For example, the safety criticalhardware may be coupled to a brake system of a train operating on therailway system. Further, the system 100 may be capable of beingcertified according to an industry recognized safety standard.

The safety critical hardware receives data elements from the safetycritical software to actuate a brake of the brake system. The system 100interfaces with the safety critical hardware and software to ensure thesafety critical hardware and software are operating according to apredetermined operating standard. It is understood that while only abrake system of the train is described, the principles of the presentdisclosure apply to any safety critical hardware and software. Otherpossible applications for the embodiments described herein include, butare not limited to, components of an airline system, components of amedical treatment system, components of an oil and gas control system,components of a smart grid system, and components of variousmanufacturing systems.

In some implementations, the system 100 receives a plurality of incomingdata packets from an external safety system, such as the railway system.The system 100 is configured to process the plurality of incoming datapackets and communicate a plurality of outgoing data packets to safetyrelevant components of the external safety system. For example, thesystem 100 determines whether a first packet of the plurality ofincoming data packets is a valid packet. When the system 100 determinesthe first packet is a valid packet, the system 100 communicates anoutgoing packet to at least one safety relevant component of the railwaysystem.

The first packet includes data elements to be acted on by the at leastone safety relevant component of the railway system. The data elementsmay include sensor data and/or input/output (I/O) point states. The atleast one safety relevant component may be a brake coupled to a trainriding on the railway system. It is understood that while only safetyrelevant components of the external safety system are described, thefirst packet may include data elements to be acted on by non-safetyrelevant components of the external safety system. The data elements areformatted according to a transfer protocol. For example, the railwaysystem is configured to package the data elements according to apredetermined packaging standard into transferable packets. The railwaysystem then transfers the plurality of incoming data packets accordingto the transfer protocol.

The system 100 is arranged to receive packets transmitted according tothe transfer protocol. Further, the system 100 is configured tointerpret the predetermined packaging standard. The system 100 thenextracts the data elements from the first packet and generates anoutgoing data packet based on the data elements. The outgoing datapacket includes a set of instructions based on the data elements. Whileonly instructions are discussed, the outgoing data packet may alsoinclude operating instruction to control I/Os, a request to read aninput in order to gather information, health message communications, arequest for inter-process communication, or other suitable elements. Theset of instructions includes at least one instruction that instructs atleast one of the safety critical hardware and software to execute aprocedure.

For example, the set of instructions may instruct the safety criticalsoftware to execute a braking procedure. The braking procedure includeshardware braking instructions. The hardware braking instructions arecommunicated to the safety critical hardware. The safety criticalhardware executes the braking instructions. For example, the safetycritical hardware applies a brake.

The system 100 determines whether to communicate the outgoing datapacket and data elements to the safety critical hardware and software.For example, the system 100 ensures each of the plurality of incomingdata packets meet a predetermined safety standard. The predeterminedsafety standard includes determining whether or not the railway systemis operating according to a predefined set of operating standards. Thesystem 100 verifies that each of the plurality of incoming data packetswas intentionally transferred by the railway system 100. For exampleonly, the railway system may transfer erroneous incoming data packetsdue to a hardware or software fault within the railway system.

The safety critical hardware and software receives the first packet ofthe plurality of incoming data packets in response to a command from anoperator of the railway system. The safety critical hardware andsoftware receives a second packet of the plurality of incoming datapackets due to a fault in the railway system. The fault in the railwaysystem may include, by way of non-limiting example only, a hardwarefailure such as a deteriorated electrical connection due to extendedexposure to heat or moisture. The safety critical hardware and softwarecommunicate the plurality of incoming data packets, including the firstand second packet, to the system 100. The system 100 is configured todetermine whether each of the plurality of incoming data packets wasreceived by the safety critical hardware and software as a result of afault in the railway system.

When the system 100 determines one of the plurality of incoming datapackets was received in response to a command from the operator, thesystem 100 generates an outgoing data packet corresponding to thereceived incoming data packet. For example, the system 100 generates afirst outgoing data packet based on the first packet. The first outgoingdata packet includes a set of instructions corresponding to the dataelements within the first packet. When the system 100 determines thefirst packet is a valid packet, the system 100 communicates the firstoutgoing data packet to the safety critical hardware and software. Forexample, the system 100 determines the first packet was received inresponse to a command from the operator. The system 100 communicates thefirst outgoing data packet to the safety critical hardware and software.The safety critical hardware and software execute the set ofinstructions included in the first outgoing data packet.

Conversely, when the system 100 determines one of the plurality ofincoming data packets was received in response to a fault within therailway system, the system 100 does not communicate an outgoing datapacket to the safety critical hardware and software. For example, thesystem 100 determines the second packet was received due to a fault inthe railway system. The system 100 does not communicate an outgoing datapacket corresponding to the second packet to the safety criticalhardware and software. Consequently, the safety critical hardware andsoftware do not execute instructions corresponding to data elementsincluded in the second packet.

Further, the system 100 generates a fault indication based on thedetermination that a fault occurred within the railway system. In thisway, data elements executed by the safety critical hardware and softwareare first verified by the system 100. This verification ensures therailway system is operating according to the predetermined safetystandard.

In some implementations, the system 100 receives a first packet of theplurality of incoming data packets. Simultaneously, the system 100receives a second packet of the plurality of incoming data packets. Thesystem 100 then executes voting logic on the first and second packet.The voting logic may be implemented as a dual two-of-two (2oo2) system.The 2oo2 voting logic is explained in greater detail below. The system100 determines whether the first and second packets are identical. Whenthe system 100 determines the first and second packets are identical,the system 100 generates a first outgoing data packet and communicatesthe first outgoing data packet to at least one component of the safetycritical hardware and software.

The at least one component then executes operating data elementsincluded within the first outgoing data packet. Conversely, when thefirst and second packets are not identical, the system 100 identifies atleast one component of the system 100 or the railway system as faulty.It is understood that while a railway system is described, theprinciples of the present disclosure apply to any external safetysystems.

The system 100 also generates a safety indication. The safety indicationmay be indicative of a failure within the system 100 or the safetycritical hardware and software. Further, the system 100 instructs the atleast one component to operate in a predetermined safe state. Forexample, the safe state may include a set of safe state data elementsarranged to maintain a safe operating environment of the railway system.

The safe state data elements include instructing the railway system tooperate in a predetermined operating mode that ensures the overallsafety of the railway system. For example only, the predeterminedoperating mode includes bringing a train operating on the railway systemto a stop. In some implementations, the safe state includes disablingall safety relevant communication interfaces. For example, a fail-safecomputer operating in a safe state is incapable of communicating withthe safety critical hardware and software. In this way, the fail-safecomputer operating in the safe state is incapable of incorrectlyinstructing the safety critical hardware and software.

The system 100 includes an active fail-safe chassis (FSC) 104 and astandby FSC 108. In order to increase availability and reliability ofthe system 100, the active FSC 104 and the standby FSC 108 are redundantFSCs. For example, the active FSC 104 is configured to execute any andall operations of the standby FSC 108. In this way, when one of theactive FSC 104 and the standby FSC 108 encounters a hardware or softwarefailure, the other of the active FSC 104 and the standby FSC 108 isconfigured to operate in place of the failed FSC.

The active FSC 104 implements a two-out-of-two (2oo2) votingarchitecture that detects voting mismatches and performs a failsafeoperation when a voting mismatch occurs. The 2oo2 voting architectureincludes dual redundant processing and voting subsystems. The redundantprocessing and voting subsystems vote on packets entering or leaving theactive FSC 104. For example, the active FSC 104 receives a plurality ofincoming data packets. The active FSC 104 receives two copies of a firstpacket of the plurality of incoming data packets.

The active FSC 104 determines the validity of the first packet. Theactive FSC 104 continuously generates a first health status signal and asecond health status signal based on the determination of whether thefirst packet is valid. In some implementations, continuously generatinga signal may include setting the signal to a first predetermined value.The continuously generated signal is then maintained at the firstpredetermined value until the signal is de-asserted to a secondpredetermined value.

The active FSC 104 compares each of the two copies of the first packet.When the two copies are identical, the active FSC 104 determines thefirst packet is valid. The active FSC 104 continuously generates thefirst health status signal and the second health status signal when theactive FSC 104 determines the first packet is valid. The first andsecond health status signals may be asserted to a first value. In someimplementations, the first value is equal to 1. In anotherimplementation, the first value may be a string of characters thatindicate the FSC 104 is healthy.

Conversely, when the active FSC 104 determines the two copies of thefirst packet are not identical, the active FSC 104 de-asserts the firstand second health status signals. It is understood that the first andsecond health status signals may be de-asserted to a second value. Forexample, the second value may be equal to 0. In another implementation,the second value may be a string of characters that indicate the FSC 104is not healthy. As will be described in greater detail below, the activeFSC 104 includes a first processor and a second processor. The firstprocessor continuously generates the first health status signal when thefirst processor determines the copy of the first packet is valid.Similarly, the second processor continuously generates the second healthstatus signal when the second processor determines the second copy ofthe first packet is valid.

The active FSC 104 communicates the first and second health statussignals to the system 100. The system 100 monitors the first and secondhealth status signal to determine whether the active FSC 104 is healthy.For example, when the system 100 receives the first and second healthstatus signals, the system 100 determines the active FSC 104 is healthy.Conversely, when the system 100 does not receive one of the first healthstatus signal and the second health status signal, the system 100determines the active FSC 104 is not healthy.

As will be discussed in detail below, the system 100 instructs theactive FSC 104 to operate in a safe state in order to maintain thesafety of the system 100. It is understood that while only votingmismatches are discussed in relation to the first and second statussignals being de-asserted, any hardware or software failure within thesystem 100 may cause one of the first and second status signals to bede-asserted. The health status signal may indicate the system 100 isoperating according to a predetermined operating standard. In this way,the health status signal indicates the system 100 is healthy.

When the active FSC 104 determines that the two copies of the firstpacket are not identical, the active FSC 104 determines a voting faulthas occurred. The active FSC 104 generates a fault signal based on thedetermination that a voting fault has occurred. The fault signal isindicative of a detected fault within the system 100. The fault may bedue to hardware or software failure within the system 100 or theexternal safety system. For availability reasons, if the active FSC 104encounters a hardware or software failure, the system 100 forces theactive FSC 104 to a predetermined safe state. The standby FSC 108 thenoperates as the active FSC.

Each of the active FSC 104 and the standby FSC 108 may include aplurality of communication links. The communication links include, butare not limited to, uplinks, downlinks, and a user interface. An uplinkis a bi-directional communication channel to higher-level equipment. Thehigher-level equipment may be system hardware included in an overallsystem architecture. For example, the higher-level equipment may becomponents of a railway braking system of the railway system.

A downlink is a bi-directional communication channel to lower-levelhardware. For example, the lower-level hardware may include switches andrelays that communicate with the higher-level equipment. The downlinksmay be implemented following any suitable hardware standard. Forexample, the downlinks may be implemented as RS-232, RS-422, RS-485,CAN, MVB, Ethernet, Ethernet HSR Ring, or other suitable technology. Auser interface is an interface designed to allow a user of the system100 to access the components and subsystems of the system 100. Forexample only, the user interface may implemented as a plurality of lightemitting diodes (LEDs).

The communication links communicate with hardware components andsoftware applications external to the system 100. Further, the system100 is configured to receive input and communicate output to a fieldengineer. For example, the field engineer may selectively configure thesystem 100 by communicating with the a user interface of the system 100.

Each of the active FSC 104 and the standby FSC 108 are arranged toexecute a health and safety application. For example, the active FSC 104and the standby FSC 108 include a memory and a processor. The health andsafety application may be software loaded into the memory of the activeFSC 104 and the standby FSC 108. The health and safety application isthen executed by the processor of the active FSC 104 and the standby FSC108.

The health and safety application monitors and controls the active FSC104 and the standby FSC 108. For example, the health and safetyapplication generates a plurality of health signals. The plurality ofhealth signals may include, but is not limited to, a module healthsignal and a mask signal. The health and safety application generatesthe plurality of signals based on a fault in active FSC 104, forexample. The health and safety application is a functional safetysubsystem designed to detect and respond to failures by forcing one ofthe active FSC 104 or the standby FSC 108 to the safe state.

It is understood that the health and safety application may include anysafety applications executed on an external safety system. This mayinclude ensuring the validity and safety of a set of data elements priorto the set of data elements being executed by the hardware and softwareof the railway system. The health and safety application is alsoconfigured to ensure the railway system operates in the predeterminedsafe state when the health and safety application detects a failure inthe system 100.

Each of the active FSC 104 and the standby FSC 108 includes a processingsubsystem. The active FSC 104 processing subsystem includes a mastercentral processing unit (CPU) 116 a and a checker CPU 120 a. Similarly,the standby FSC 108 processing subsystem includes a master CPU 116 b anda checker CPU 120 b. Each of the master CPUs 116 a and 116 b areconfigured to be identical. Similarly, each of the checker CPUs 120 aand 120 b are configured to be identical. In this way, the master CPU116 a, the master CPU 116 b, the checker CPU 120 a, and the checker CPU120 b act to increase the availability and reliability of the system100.

In order to monitor and control the overall health and safety of thesystem 100, the master CPU 116 a and the checker CPU 120 a are alsoconfigured to be identical. For example, the master CPUs 116 a and thechecker CPU 120 a execute identical software and implement identicalprocessing functions. The master CPU 116 a and the checker CPU 120 a areconfigured to perform identical functions in order to maintain fullhardware fault and error detection coverage of the safety criticalhardware and software and allow for voting on outgoing packets in thecorrect sequence.

In some implementations, the master CPU 116 a receives a request toperform a task from at least one of the safety critical hardware orsoftware of the railway system. The request to perform a task is basedon a command received by the safety critical hardware and software fromthe operator of the external safety system. For example, the operator ofthe railway system may command a train riding on the railway system toincrease speed. The operator may actuate a physical lever within thetrain. The physical lever is arranged to generate a signal indicative ofthe command to increase the speed of the train. The signal iscommunicated to the safety critical hardware and software.

Prior to executing the command, the safety critical hardware andsoftware communicate the safety relevant task to the system 100. Thesystem 100 is configured to interpret the signal and determine arequested task associated with the signal. The system 100 thendetermines whether the requested task is indicative of a safety relevanttask. For example, the requested task may be a safety relevant task or anon-safety relevant task. In one example. a non-safety relevant taskincludes instructions that instruct the safety critical hardware andsoftware to return a current status of the train operating within therailway system. In another example, a safety-relevant task includesinstructions that instruct the safety critical hardware and software toincrease the speed of the train.

The system 100 compares the requested task to a list of predefinedtasks. The system 100 determines, based on the predefined tasks, whetherthe requested task is safety relevant. When system 100 determines therequested task is a safety relevant task, the system 100 generates apacket corresponding to the requested task. It is understood that whileonly a safety relevant task is described herein, the system 100 maygenerate a packet from any task. The system 100 communicates the packetto the master CPU 116 a. Both the master CPU 116 a and the checker CPU120 a are configured to verify the validity of the packet before thesafety critical hardware and software execute the requested task. Forexample, the request to perform a task may be one of the plurality ofincoming data packets.

Simultaneously, the checker CPU 120 a receives a copy of the same packetcorresponding to the requested task. Each of the master CPU 116 a andthe checker CPU 120 a is configured to communicate a copy of the packetto the other of the master CPU 116 a and the checker CPU 120 a. Prior tothe master CPU 116 a instructing the safety critical hardware andsoftware to perform the requested task, the master CPU 116 a and thechecker CPU 120 a compare the packet received by each of the master CPU116 a and the checker CPU 120 a. For example, the master CPU 116 a andthe checker CPU 120 a perform 2oo2 voting on the packet.

When the packets are identical, the master CPU 116 a instructs thesafety critical hardware and software to perform the requested task. Forexample, the master CPU 116 a and the checker CPU 120 a generates anoutgoing packet associated with the requested task. When voting issuccessful (i.e., the packets are identical) the master CPU 116 acommunicates the outgoing packet to the safety critical hardware andsoftware to execute the requested task.

The outgoing packet may include operating data elements to control I/Os,a request to read an input in order to gather information, healthmessages, a request for inter-process communication, or other suitablerequests associated with the safety critical hardware and software. Theoperating data elements may include a set of instructions that instructone of the safety critical hardware and software to execute apredetermined procedure. For example, the operating data elements mayinstruct the safety critical software to execute an accelerationprocedure. The acceleration procedure includes controlling at least onesafety critical hardware component. The at least one hardware componentincludes a throttle mechanism of the train.

Conversely, when the packets are not identical, the master CPU 116 a andthe checker CPU 120 a do not instruct the safety critical hardware andsoftware to perform the requested task. The master CPU 116 a generates afault signal. The fault signal is indicative of a failure within thesystem 100, the safety critical hardware and software, or the externalsafety system. In this way, the master CPU 116 a verifies that therequested task is a valid request from a valid source (i.e., an externalhardware component or software application.)

For example only, the master CPU 116 a receives erroneous requests toperform tasks due to external hardware failure, a software error, or anoverall system malfunction. It is understood that while only the masterCPU 116 a and the checker CPU 120 a are described, the principlesdescribed above also apply to the master CPU 116 b and the checker CPU120 b.

Each of the active FSC 104 and the standby FSC 108 include a pluralityof input/output unit (IOU) modules. The active FSC 104 includes IOUmodules 128 a-128 f. The IOU modules 128 a-128 f will hereinafter bereferred to as the IOU module 128. Similarly, the standby FSC 108includes IOU modules 132 a-132 f, which are identical to the IOU modules128 a-128 f. The IOU modules 132 a-132 f will hereinafter be referred toas the IOU module 132.

The active FSC 104 and the standby FSC 108 also each include a votingsubsystem, an input/output (I/O) subsystem, and a switch module 124 aand a switch module 124 b respectively. The switch module 124 a and theswitch module 124 b are configured to be identical. The switch module124 a and the switch module 124 b route packets received from the I/Osubsystem to the master CPUs 116 a and 116 b and the checker CPUs 120 aand 120 b.

For example, the switch module 124 a may route a packet received fromthe master CPU 116 a to an external network component. Further, theswitch module 124 a receives packets from an external component androutes the received packets to the master CPU 116 a, the checker CPU 120b, and the I/O subsystem. Further, the switch module 124 a communicatesthe received packets to the voting subsystem. It should be appreciatedthat while only the switch module 124 a is described, the principlesdescribed above also apply to the switch module 124 b.

The IOU module 128 implements the I/O subsystem within the active FSC104. The I/O system is configured to provide a gateway betweenlower-level network equipment and the master CPU 116 a and the checkerCPU 120 a. The IOU module 128 isolates the master CPU 116 a and thechecker CPU 120 a from low level protocols required by external hardwareand software. In this way, the IOU module 128 adds a safety layer on theinternal Ethernet to support safe communication between the master CPU116 a, checker CPU 120 a, and the external hardware and software. TheIOU module 128 and the switch module 124 a may be referred tocollectively as the front blades 124 a-128 f. Similarly, the IOU module132 and the switch module 124 b may be referred to collectively as thefront blades 124 b-132 f.

The active FSC 104 and the standby FSC 108 also include a plurality ofrear transition modules (RTMs). The active FSC 104 includes RTMs 136a-136 i. The standby FSC 108 includes RTMs 140 a-140 i. It should beappreciated that because the standby FSC 108 is a redundantconfiguration of the active FSC 104, the RTMs 136 a-136 i are identicalto the RTMs 140 a-140 i. In some implementations, each of the RTMs 136a-136 i may be hot swappable field replaceable units (FRUs).

For example, each of the RTMs 136 a-136 i may be removed and replaced bya field engineer without first having to power off the entire system100. The RTMs 136 a-136 i allows the active FSC 104 to communicate withthe external hardware and software components. For example, the RTM 136a electrically and logically couples the master CPU 116 a to componentsexternal to the active FSC 104. Similarly, the RTM 136 b electricallyand logically couples the standby FSC 108 to external hardware andsoftware components.

The front blades 124 a-128 f each includes a first memory location and asecond memory location. The first and second memory locations store avalue received from a corresponding hardware component of the active FSC104. For example, the master CPU 116 a continuously generates a modulehealth signal (described above) indicative of the module health of themaster CPU 116 a. The master CPU 116 a stores a 1 in the first memorylocation when the module health signal indicates the master CPU 116 a ishealthy.

Each of the front blades 124 a-128 f and the master CPU 116 a generatesa module health signal. Each of the front blades 124 a-128 f stores avalue indicative of the module health signal in a first memory location.For example only, the switch module 124 a stores a 1 in a first memorylocation associated with switch module 124 a when the module healthsignal indicates the switch module 124 a is healthy.

Conversely, the switch module 124 a stores a 0 in the first memorylocation when the module health signal indicates the switch module 124 ais not healthy. It is understood that while only the switch module 124 ais described, the principles discussed herein apply to all moduleswithin the system 100. It is further understood that while only storinga 1 and 0 based on the module health signal are described, any suitableindicator may be used to indicate the health of each of the moduleswithin the system 100. Each of the modules within the system 100 thencommunicates the value stored in the associated first memory location tothe master CPU 116 a.

The master CPU 116 a determines whether the active FSC 104 is healthybased on receipt of a plurality of values corresponding to the modulehealth signals generated by each of the front blades 124 a-128 f and amodule health signal generated by the master CPU 116 a. For exampleonly, the master CPU 116 a determines that the active FSC 104 is healthywhen each of the received values corresponding to the module healthsignals indicates that each of the modules within the system 100 ishealthy. Conversely, the master CPU 116 a determines that the active FSC104 is not healthy when at least one of the received valuescorresponding to the module health signals indicates that at least oneof the modules within the system 100 is not healthy.

In another implementation, a value indicative of the module healthsignal is stored in a module configured at the front of the system 100.For example, a module configured on the opposite side of the system 100from the RTM 136 s stores a 1 in the first memory location when themodule health signal indications the master CPU 116 a is healthy. It isunderstood that while only the master CPU 116 a is described, theprinciples described herein apply to the checker CPU 120 a.

The master CPU 116 a receives a first packet of the plurality ofincoming data packets. Simultaneously, the checker CPU 120 a receives acopy of the first packet. The master CPU 116 a and the checker CPU 120 adetermine whether the first packet and the copy of the first packet areidentical. When the first packet and the copy of the first packet areidentical, the master CPU 116 a continuously generates a module healthsignal. The module health signal may be set to 1 or a high signal value.The value of the module health signal is indicative of the health of theactive FSC 104. The master CPU 116 a stores a 1 in the first memorylocation.

Conversely, when the first packet and the copy of the first packet arenot identical, the master CPU 116 a generates an invalid module healthsignal. For example, the master CPU 116 a de-asserts the module healthsignal. The master CPU 116 a stores a 0 in the first memory location.The master CPU 116 a receives a mask signal indicative of the state ofthe mask register within the master CPU 116 a. For example, the maskregister is a programmable bit within each of the master CPU 116 a andthe checker CPU 120 a.

The mask register is set when a requested task is a safety-relevanttask. For example, the mask register is set to 1 when the system 100determines the requested task is a safety-relevant task. The RTM 136 astores a 1 in the second memory location when the mask signal indicatesthe mask register within the master CPU 116 a is set to a 1. Conversely,the RTM 136 a stores a 0 in the second memory location when the masksignal indicates the mask register within the master CPU 116 a is set toa 0.

The system 100 also includes a watchdog timer application. The watchdogtimer application is a continuously running watchdog function thatindicates whether the master CPU 116 a is has stopped executing code.For example, the watch dog timer application monitors the master CPU 116a. The master CPU 116 a generates a watchdog timer reset signal. Themaster CPU 116 a communicates the watchdog timer reset signal to thewatchdog timer application. The watchdog timer application determineswhether the watchdog timer reset signal was received within a predefinedperiod. For example, the predefined period may be 1 minute. Thepredefined period may be a configurable item.

The watchdog timer application generates a watchdog health signal basedon the watchdog timer reset signal. For example, when the master CPU 116a communicates a watchdog timer reset signal after the predefinedperiod, the watchdog timer application sets the watchdog health signalto 0. Conversely, when the master CPU 116 a communicates the watchdogtimer reset signal within the predefined period, the watchdog timerapplication sets the watchdog health signal to 1. In other words, thewatchdog timer application determines a fault in the master CPU 116 awhen the master CPU 116 a is incapable of communicating with thewatchdog application within the predefined period. For example, thefault may be as a result of the master CPU 116 a being stuck in asoftware loop. It is understood that while the only master CPU 116 a isdescribed above, the checker CPU 120 a is configured to identical tasksas the master CPU 116 a.

The RTM 136 a communicates a module health value indicative of theoverall health of the master CPU 116 a. For example, the RTM 136 a isconfigured to perform a logical OR on the value stored in the firstmemory location and the value stored in the second memory location. Forexample, when the value stored in the first memory location is 0 andwhen the value stored in the second memory location is 0 the RTM 136 awill a communicate a module health value of 0.

When the value stored in the first memory location is 1 and the valuestored in the second memory location is a 0, the RTM 136 a communicatesa module health value of 1. It is understood that each of the RTMs 136a-136 i is configured to perform a logical OR on values stored withinmemory of associated with each of the front blades 124 a-128 f.

The RTMs 136 a and 136 b communicate a chassis health value to thesafety relay box module 144. The safety relay box module 144 selectivelydetermines which of FSCs within the system 100 will be the active FSCand which will be the standby FSC based on a plurality of chassis healthvalues. For example, the safety relay box module 144 receives aplurality of chassis health values from the RTM 136 a, RTM 136 b, RTM140 a, and RTM 140 b. The RTM 136 a generates a first chassis healthvalue based on the cumulative result of the module health signalsgenerated by the master CPU 116 a and the front blades 124 a-128 f and avalue stored in the mask registers.

The RTM 136 a communicates the chassis health value to the safety relaybox module 144. Similarly, the RTM 136 b generates a second chassishealth value based the cumulative result of the module health signalsgenerated by the checker CPU 120 a and the front blades 124 a-128 f anda value stored in the mask registers. The RTM 140 a generates a thirdchassis health value based on the cumulative result of the module healthsignals generated by the master CPU 116 b and the front blades 124 b-132f and a value stored in the mask registers. Similarly, the RTM 140 bgenerates a fourth chassis health value based on the cumulative resultof the module health signals generated by the checker CPU 120 b and thefront blades 124 b-132 f and a value stored in the mask registers.

The safety relay box module 144 determines which of the active FSC 104and the standby FSC 108 will operate as an active FSC within the system100. For example, the safety relay box module 144 determines the activeFSC 104 will operate as the active FSC within the system 100 based thefirst-fourth chassis health values. The safety relay box module 144determines the active FSC 104 will be the active FSC within the system100 when the first and second chassis health signals indicate the activeFSC 104 is healthy.

The safety relay box module 144 generates an active FSC signal and astandby FSC signal based on the active FSC determination. For example,the safety relay box module 144 generates a first active FSC signal anda second active FSC signal. The first and second active FSC signals areredundant signals that are both indicative of which FSC is elected tooperate as the active FSC. The safety relay box module 144 alsogenerates a first standby FSC signal and a second standby FSC signal.

The first and second standby FSC signals are redundant signals that areboth indicative of which FSC was elected to operate as the standby FSC.The safety relay box module 144 communicates the first active FSC signaland the first standby FSC signal to the active FSC 104. Similarly, thesafety relay box module communicates the second active FSC signal andthe second standby FSC signal to the standby FSC 108.

The safety relay box module 144 also determines whether a fault hasoccurred in one of the active FSC 104 and the standby FSC 108 based onfirst-fourth chassis health signals. For example, the safety relay boxmodule 144 determines a fault has occurred in the active FSC 104 whenthe safety relay box module 144 receives an invalid chassis healthsignal from one of the RTMs 136 a and 136 b. For example, an invalidchassis health signal may be a chassis health value of 0. The safetyrelay box module 144 generates a standby signal based on the invalidchassis health signal. The safety relay box module 144 communicates thestandby signal to the active FSC 104. The active FSC 104 determineswhether to enter the failsafe state based on the standby signal. Forexample, the active FSC 104 determines to enter the failsafe state whenthe active FSC 104 receives the standby signal.

The safety relay box module 144 generates an active signal based on thechassis health value. The safety relay box module 144 communicates theactive signal to the standby FSC 108. The active signal indicates to thestandby FSC 108 that the active FSC 104 is operating in the failsafestate or standby state. When the standby FSC 108 receives the activesignal, the standby FSC 108 begins operating as the active FSC. In thisway, the safety relay box module 144 forces a failed active FSC into afailsafe state, and alerts a standby FSC to assume the role of activeFSC.

In some implementations, the safety relay box module 144 is configuredto communicate with a plurality of commercial off the shelf (COTS)products. For example, the active FSC 104 and the standby FSC 108 may beCOTS products implemented within the system 100. The active FSC 104 andthe standby FSC 108 are configured according to a predeterminedimplementation standard. The implementation standard includes apredefined communication protocol.

The safety relay box module 144 is configured to receive and interpretsignals and messages from the active FSC 104 and the standby FSC 108according to the communication protocol. In this way, the safety relaybox module 144 communicates with the active FSC 104 and the standby FSC108 regardless of the physical characteristics of the active FSC 104 andthe standby FSC 108 so long as the active FSC 104 and the standby FSC108 follow the communication protocol. It is understood that the safetyrelay box module 144 communicates with any component within the system100. Further, any component within the system 100 may be implementedutilizing COTS products so long as the any component follows thecommunication protocol.

In some implementations, the I/O subsystem also receives packets fromexternal sources such as hardware and software components of the railsystem. The received packets may include the plurality of incoming datapackets as discussed above. The I/O subsystem then communicates thereceived packets to the voting subsystem.

The voting subsystem is a functional safety subsystem and is configuredto detect and respond to failures. When the voting subsystem detects anerror in one of the active FSC 104 and the standby FSC 108, the votingsubsystem forces the one of the active FSC 104 and the standby FSC 108to the failsafe state. The voting subsystem performs fault detection ofthe processing subsystem. For example, the master CPU 116 a and thechecker CPU 120 a generate outgoing packets based on a requested task.

The voting subsystem compares the outgoing packets generated by themater CPU 116 a to the outgoing packets generated by the checker CPU 120a. When the outgoing packets are identical, the voting subsystemcommunicates a single copy of a packet to the switch module 124 a and tothe I/O subsystem. When the outgoing packets are not identical, thevoting subsystem forces the active FSC 104 to the failsafe state. Inthis way, the voting subsystem detects faults in one of the active FSC104 and the standby FSC 108. Upon detecting a fault in the one of theactive FSC 104 and the standby FSC 108, the voting subsystem brings thesystem 100 to a safe state by forcing the faulty FSC to the failsafestate.

The voting subsystem also receives packets from the I/O subsystem. Forexample, the IOU modules 128 communicate with the voting subsystem viathe switch modules 124 a. The IOU modules 128 communicate a first packetto switch module 124 a. The switch module 124 a duplicates the firstpacket. For example, the switch module 124 a generates a second packet.The second packet is a copy of the first packet. The switch module 124 acommunicates the first packet to the master CPU 116 a and the checkerCPU 120 a. The master CPU 116 a and the checker CPU 120 a executingvoting logic on the first and second packets as described above. In thisway, the voting subsystem verifies the validity of the received packets.

In another example, the first packet may not be identical to the secondpacket due to a hardware malfunction in an external component, such asthe safety critical hardware of the rail system. When the votingsubsystem determines the first packet is not identical to the secondpacket, the safety application forces the active FSC 104 to the failsafestate. Additionally or alternatively, the mismatched packets may besilently discarded. For example, neither the master CPU 116 a nor thechecker 120 a generates an outgoing packet based on the first and secondpacket. In this way, the system 100 may avoid excessivefailover/failsafe transitions due to otherwise recoverable errors. Inanother implementation, input voting may be configured to result infailover/failsafe transition.

Referring now to FIG. 2 a block diagram of an exemplary fail-safechassis (FSC) according to the principles of the present disclosure isshown at 200. The FSC 200 includes a master CPU 204 and a checker CPU208. The master CPU 204 and the checker CPU 208 execute identicalsoftware and implement identical processing functions. The master CPU204 and the checker CPU 208 are configured to perform identicalfunctions in order to maintain full hardware fault and error detectioncoverage of safety critical hardware and software.

For example, the master CPU 204 receives a request to perform a taskfrom at least one of the safety critical hardware or software of anexternal safety system. The external safety system may be a system thatis required to operate according to a predetermined safety standard,such as a railway system. The railway system includes safety criticalhardware and software. The safety critical hardware and softwareinteract with components of the railway system in order to controlaspects of the railway system. For example, the safety critical hardwareand software controls a train operating within the railway system.

The request to perform a task may include, but is not limited to,determining a status of a hardware component, engaging a brake of therailway system, or indicating to an operator of the railway system alocation of a train operating within the railway system. The request toperform a task is received in the form of a packet. For example, therequest to perform a task may be one of the plurality of incoming datapackets.

Simultaneously, the checker CPU 208 receives a copy of the same request.Each of the master CPU 204 and the checker CPU 208 is configured tocommunicate a copy of the requested task to the other of the master CPU204 and the checker 208. Prior to the master CPU 204 performing therequested task, the master CPU 204 and the checker CPU 208 compare therequested task received by each of the master CPU 204 and the checkerCPU 208.

When the requested tasks are identical, the master CPU 204 and thechecker CPU 208 communicate the requested task to the safety criticalhardware and software. For example, the master CPU 204 and the checkerCPU CPU 208 generates an outgoing packet associated with the requestedtask. The outgoing packet includes operating data elements. Theoperating data elements may include a set of instructions that instructone of the safety critical hardware and software to execute apredetermined procedure. For example, the operating data elements mayinstruct the safety critical software to execute a brake engagingprocedure. The brake engaging procedure includes controlling at leastone safety critical hardware component. The at least one hardwarecomponent may include a brake mechanism of the railway system.

Conversely, when the requested tasks are not identical, the master CPU204 and the checker CPU 208 do not communicate the requested task to thesafety critical hardware and software. In this way, the master CPU 204verifies that the requested task is a valid request from a valid source(i.e., an external hardware component or software application.) Forexample only, the master CPU 204 receives erroneous requests to performtasks due to external hardware failure, a software error, or an overallsystem malfunction.

The master CPU 204 and the checker CPU 208 include a mask register. Themask register is a programmable bit within the master CPU 204 and thechecker CPU 208. A value stored in the mask register is indicateswhether a requested task is a safety-relevant task. For example, themask register is set to 1 when the system 100 determines a requestedtask is a safety-relevant task. Further, the health and safetyapplication sets the mask register when the system 100 is initiallystarted.

The master CPU 204 and checker CPU 208 each continuously generate amodule health signal based on a determination of whether the requeststasks are identical. When the master CPU 204 and the checker CPU 208determine the requested tasks are identical, the master CPU 204 and thechecker CPU 208 continuously generate the module health signal pulse.When the master CPU 204 and the checker CPU 208 determine the requestedtasks are not identical, the master CPU 204 and the checker CPU 208de-assert the module health signal pulse.

The master CPU 204 and the checker CPU 208 generate a chassis healthsignal pulse based on a plurality of module health signal pulsegenerated by each of a plurality of modules associated with the FSC 200.The FSC 200 includes similar features as those described with respect tothe active FSC 104. For example, the FSC 200 includes the front blades124 a-128 f as described above. Each of the modules within the FSC 200continuously generates a module health signal (as described above). Themaster CPU 204 generates a first chassis health signal pulse when theplurality of module health signals indicates the FSC 200 is healthy. Inother words, the first chassis health signal pulse is continuouslyasserted (i.e., set to 1) when the mask register indicates the requestedtask is safety relevant and the plurality of module health signalsindicate the FSC 200 is healthy.

It is understood that when the mask register indicates the requestedtask is not safety relevant (i.e., the mask register is set to 0) thefirst module health signal pulse may be de-asserted. However, safetyrelevant procedures (i.e., forcing the active FSC 204 to a safe state)will not be executed. In this way, failover/failsafe procedures are onlyimplemented when a fault occurs during a safety relevant task. Thechassis health signal pulse is indicative of the overall health of theFSC 200.

It is understood that principles described with respect to the masterCPU 204 apply to the checker CPU 208. The checker CPU 208 generates asecond chassis health signal pulse based on a plurality of module healthsignal pulse generated by each of a plurality of modules associated withthe FSC 200. The checker CPU 208 continuously generates the secondchassis health signal pulse when the plurality of module health signalsindicates the FSC 200 is healthy. The master CPU 204 communicates afirst chassis health signal pulse to a safety relay box 212. Similarly,the checker CPU 208 communicates a second chassis health signal pulse toa safety relay box 212.

The safety relay box module 212 selectively determines whether the FSC200 will operate as an active FSC based on a plurality of chassis healthsignals. For example, the safety relay box module 212 begins receiving achassis health signal pulse from the master CPU 204 when the FSC 200 isinitiated. The safety relay box 212 also begins receiving a chassishealth signal pulse from the checker CPU 208. The safety relay boxmodule 212 determines whether the FSC 200 will be elected as the activeFSC based on the chassis health signals.

In some implementations, the safety relay box module 212 also receives aplurality of chassis health signal pulses from another FSC. The safetyrelay box module 212 determines whether the FSC 200 or the other FSCwill be elected as the active FSC based on the plurality of chassishealth signal pulses. For example, the safety relay box module 212 firstdetermines whether one of the FSC 200 or the other FSC is healthy. Thesafety relay box module 212 determines the FSC 200 is healthy when thechassis health signal pulses received from the FSC 200 indicate the FSC200 is healthy. For example, the safety relay box 212 determines the FSC200 is healthy when the safety relay box 212 receives a chassis healthsignal from the FSC 200.

Similarly, the safety relay box module 212 determines the other FSC ishealthy when the plurality of chassis health signal pulses received fromthe other FSC indicate the other FSC is healthy. When safety relay boxmodule 212 determines that both of the FSC 200 and the other FSC arehealthy, the safety relay box module 212 elects an active FSC based onwhich of the FSC 200 and the other FSC communicated one of the pluralityof chassis health signal pulses first. For example, when the FSC 200communicates a signal to the safety relay box module 212 before theother FSC communicates a signal to the safety relay box module 212, thesafety relay box module 212 elects the FSC 200 as the active FSC.

The safety relay box module 212 monitors the health of the active FSC.For example, the safety relay box module 212 receives the first andsecond chassis health signal pulses from the FSC 200. The safety relaybox module 212 determines whether a fault has occurred in the FSC 200based on the first and second chassis health signal pulses. The safetyrelay box 212 determines a fault has occurred in the FSC 200 when atleast one of the first and second chassis health signal pulses isindicative of a mismatch of the requested tasks. In other words, thesafety relay box module 212 determines a fault has occurred when thesafety relay box module 212 does not receive one of the first and secondchassis health signal pulses.

The safety relay box module 212 generates a failsafe signal when thesafety relay box module 212 determines a fault has occurred in the FSC200. The safety relay box module 212 communicates the failsafe signal tothe FSC 200. The FSC 200 determines whether to enter the safe statebased on the failsafe signal. For example, the FSC 200 enters the safestate when the FSC 200 receives the failsafe signal. Conversely, the FSC200 will continue to operate in the active FSC mode unless the FSC 200receives a failsafe signal.

Alternatively, the FSC 200 may enter a standby state when the FSC 200does not receive an active signal. The safety relay box module 212communicates the active signal when the safety relay box module 212determines the FSC 200 is to operate as the active FSC. The safety relaybox module 212 continuously communicates the active signal to the FSC200. When the safety relay box module 212 determines the FSC 200 isfaulty, the safety relay box module 212 does not communicate an activesignal to the FSC 200. The FSC 200 then operates in the standby state.

In some implementations, the plurality of chassis health signal pulsesare implemented as dynamic pulse signals to eliminate undetected, latentfailure modes. The safety relay box module 212 is configured todetermine whether a pulse signal is stuck at 0 or stuck at 1. Forexample, the safety relay box 212 receives a first pulse signal from themaster CPU 204. The safety relay box module 212 is configured to wait apredetermined period of time.

For example, the predetermined period of time may be a periodcorresponding to a time long enough for a dynamic pulse to reset. Thesafety relay box module 212 determines the first pulse signal is stuckwhen the first pulse signal has not been reset after the predeterminedperiod. The safety relay box module 212 determines a fault in the masterCPU 204 when then safety relay box module 212 determines the first pulsesignal is stuck. The safety relay box module 212 generates a failsafesignal based on determining a fault in the master CPU 204.

In some implementations, the master CPU 204 and the checker CPU 208 runin loose lockstep with one and other. In some scenarios, the master CPU204 and the checker CPU 208 are required to run in a synchronizationmode. With particular reference to FIG. 3, a functional block diagram ofa synchronization system 300 implementing the synchronization mode isshown. The synchronization mode includes first synchronizing a systemclock of each of the master CPU 204 and the checker CPU 208 and secondsynchronizing an operating system (OS) scheduler in order for the masterCPU 204 and the checker CPU 208 to run in the same sequence at a dataelement level.

Each of the master CPU 204 and the checker CPU 208 include afield-programmable gate array (FPGA) configured to executesynchronization logic. For example, the master CPU 204 includes a masterFPGA 304 and the checker CPU 208 includes a checker FPGA 308. Each ofthe master FPGA 304 and the checker FPGA 308 is configured to implementtask synchronization logic. For example, each of the master CPU 204 andthe checker CPU 208 are synchronized when a task requests a timeinstruction and when a task sends and/or receives an Ethernet packet.Further, each of the master FPGA 304 and the checker FPGA 308 may besynchronized to a reference clock.

The reference clock is a 25 megahertz (MHz) clock within the system 100.For example only, the reference clock may be a CPU decrementer. Thereference clock is a common reference clock that each module within thesystem 100 syncs with. Each module within the system 100 includes aphase locked loop that is synchronized with the reference clock. Forexample, the master CPU 204 a first clock that counts at a firstinterval. The master CPU 204 includes a first phase locked loop thatdetermines a reference point between the first interval and a referenceclock interval. When the master CPU 204 communicates with another module(for example, the checker CPU 208) within the system 100, the master CPU204 determines a current reference clock time based on the referencepoint between the first interval and the reference clock interval. Inthis manner, each module within the system 100 may reference a time onthe reference clock that is common to each module in order tocommunicate within the system 100.

Each of the master CPU 204 and the checker CPU 208 is configured toexecute a software application external to the system 100. For example,the master CPU 204 executes tasks requested by the railway applicationas described above. The master CPU 204 is configured to execute a firstcopy of the railway application (first copy). The checker CPU 208 isconfigured to execute second copy of the railway application (secondcopy). Each of the first copy and the second copy is configured toexecute tasks simultaneously. For example, when the system 100 isrestarted, each of the master CPU 204 and the checker CPU 208 initiatesthe first copy and the second copy respectively. The first copy and thesecond copy are synchronized to the reference clock in order to executetasks simultaneously.

In some implementations, the system 300 includes a scheduler module 312.The scheduler module 312 is configured to run software stored in anassociated memory. The scheduler module 312 executes an OS scheduler.The OS scheduler is a software program that communicates OS level tasksto the master CPU 204 and the checker CPU 208. In some implementations,the scheduler module 312 determines whether the master CPU 204 and thechecker CPU 208 are synchronized based on a task synchronization point.The task synchronization point may be referred to as a checkpoint.

The synchronization point may be when a task requests a timeinstruction, when each task sends and/or receives an Ethernet packet, orwhen a new task begins. Additionally or alternatively, software externalto the system 300 may control the synchronization point. The schedulermodule 312 communicates a first synchronization point to the master CPU204 and the checker CPU 208. As described above, the master CPU 204 andthe checker CPU 208 execute identical copies of application software.The master CPU 204 and the checker CPU 208 are configured to beidentical, and therefore, execute requested tasks associated with theapplication simultaneously. However, the synchronization of the masterCPU 204 and checker CPU 208 may be affected by inherent inaccuracies inthe reference clock and inherent differences in hardware tolerances.

The synchronization point is an operating system event that occurs whenthe master CPU 204 and the checker CPU 208 should be writing the sameaddress. As a non-limiting example, the master CPU 204 and the checkerCPU 208 should both write the same address after 5 cycles of thereference clock. The operating system monitors the synchronization pointof a processor module 316 within the master CPU 204 and a processormodule 320 with in the checker CPU 208.

For example, the processor module 316 and the processor module 320 eachreceive a plurality of tasks to execute from the first and second copyof the application software. In order to ensure that the processormodule 316 and the processor module 320 are executing identical tasks insynchronization, prior to executing the task, both the processor module316 and the processor module 320 write a memory address to a memoryregister. For example, the processor module 316 writes the address ofthe first synchronization point to a register A 324 when the master CPU204 reaches the synchronization point. Similarly, the processor module320 writes the address of the first synchronization point to a registerA 328 when the checker CPU 208 reaches the synchronization point.

The FPGA 304 includes a logical AND module 332 and a register B 340.Similarly, the FPGA 308 includes a logical AND module 336 and a registerB 344. The logical AND module 332 receives the address stored inregister A 324 and register A 328. The logical AND 332 performs alogical AND on the received addresses. For example only, the logical AND332 determines whether the first bit of the address in register A 324and the first bit of the address in register A 328 are the same. Whenthe first bit in both addresses are the same, the logical AND module 332writes a 1 in the first bit of the register B 340. Those skilled in theart can appreciate that while only a one bit example is described, theregister A may include a plurality of bits. Further, while only apositive logical AND outcome is described, the logical AND 332 may writea 0 to a bit within the register B 340 when the address bits in theregisters A 324 and 328 are not the same.

The logical AND module 336 receives the address stored in register A 324and register A 328. The logical AND 336 performs a logical AND on thereceived addresses. For example only, the logical AND 336 determineswhether the first bit of the address in register A 324 and the first bitof the address in register A 328 are the same. When the first bit inboth addresses are the same, the logical AND module 336 writes a 1 inthe first bit of the register B 344. Those skilled in the art canappreciate that while only a one bit example is described, the registerA may include a plurality of bits. Further, while only a positivelogical AND outcome is described, the logical AND 336 may write a 0 to abit within the register B 344 when the address bits in the registers A324 and 328 are not the same.

The scheduler module 312 monitors the register B 340 and the register B344. For example, the scheduler module 312 receives a first value fromthe register B 340 and a second value from the register B 344 at a firstsynchronization point. The first synchronization point may be apredetermined memory address. The scheduler module 312 determineswhether the first value is the same as the second value. The schedulermodule 312 determines the master CPU 204 and the checker CPU 208 aresynchronized when the scheduler module 312 determines the first value isset to 1 and the second value is set to 1. It is understood that thevalue stored in the register B 340 and register B 344 may be a series ofbits set to 1. The scheduler module 312 generates a processorsynchronization signal. The scheduler module 312 communicates theprocessor synchronization signal to the processor module 316 and theprocessor module 320.

The scheduler module 312 determines the master CPU 204 and the checkerCPU 208 are not synchronized when the scheduler module 312 determinesthe first value is not the same as the second value. For example, atleast one of the first value and the second value is not set to 1. Thescheduler module 312 increments a synchronization counter. The schedulermodule 312 receives a third value from the register B 340 and a fourthvalue from the register B 344 at a second synchronization point. Thesecond synchronization point may be after a predetermined number ofmemory addresses after the first synchronization point. The schedulermodule 312 determines whether the third value and the fourth value areset to 1.

As described above, the scheduler module 312 determines the master CPU204 and the checker CPU 208 are synchronized when the scheduler module312 determines the third value is set to 1 and the fourth value is setto 1. The scheduler module 312 generates the processor synchronizationsignal. The scheduler module 312 communicates the processorsynchronization signal to the processor module 316 and the processormodule 320. The scheduler module 312 increments the synchronizationcounter when the scheduler module 312 determines the third value is notthe same as the fourth value.

The scheduler module 312 continues to monitor values from the register B340 and the register B 344 until the scheduler module 312 determines themaster CPU 204 and the checker CPU 208 are synchronized. The schedulermodule 312 communicates the processor synchronization signal to theprocessor module 316 and the processor module 320.

The processor module 316 and the processor module 320 determine whetherto execute tasks associated with the first and second copy of theapplication software based on the processor synchronization signal. Forexample, the processor modules 316 and 320 execute tasks associated withthe first and second copies of the application when the processormodules 316 and 320 receive the processor synchronization signal. Inother words, one the scheduler module 312 determines the master CPU 204and the checker CPU 208 are synchronized, the master CPU 204 and thechecker CPU 208 are released and begin executing tasks.

Further, the processor module 316 and the processor module 320 determinewhether to send and/or receive an Ethernet packet based on the processorsynchronization signal. For example, the processor modules 316 and 320send and receive data packets associated with the system 300. The datapackets may include the plurality of data packets described with respectto FIGS. 1 and 2. By way of non-limiting example only, the processormodule 316 may generate a first outgoing data packet. The processormodule 316 transmits the first outgoing data packet when the processormodule 316 receives the processor synchronization signal. Similarly, theprocessor module 320 may generate a second outgoing data packet. Theprocessor module 320 transmits the second outgoing data packet when theprocessor module 320 receives the processor synchronization signal. Itis understood that while only transmitting data packets is described,the principles of the present disclosure also apply to each of theprocessor modules 316 and 320 receive data packets.

In some implementations, the scheduler module 312 determines whether torestart the system 300 based on the synchronization counter. Forexample, the schedule module 312 compares a value of the synchronizationcounter to a predetermined threshold. The predetermined threshold may bea value associated with a number of missed synchronization points. Forexample, the predetermined threshold may be 3. In other words, once thescheduler module 312 determines that the master CPU 204 and the checkerCPU 208 were out of synchronization 3 times. When the scheduler module312 determines the value in the synchronization counter is greater thanthe predetermined threshold, the scheduler module 312 instructs thesystem 300 to restart.

In yet other implementations, the granularity of synchronization may becontrolled based on a density of the synchronization point. For example,the scheduler module 312 compares the current memory address stored inthe register A 324 with the first synchronization point. The schedulermodule 312 increments a master sync counter when the current memoryaddress is not the same as the first synchronization point. Thescheduler module 312 continues to compare the current memory addressstored in the register A 324 with the first synchronization point. Thescheduler module 312 continues to increment the master sync counteruntil the current memory address is the same as the firstsynchronization point. In other words, the scheduler module 312 countsuntil the master CPU 204 reaches the first synchronization point.

Similarly, the scheduler module 312 compares the current memory addressstored in the register A 328 with the first synchronization point. Thescheduler module 312 increments a checker sync counter when the currentmemory address is not the same as the first synchronization point. Thescheduler module 312 continues to compare the current memory addressstored in the register A 328 with the first synchronization point. Thescheduler module 312 continues to increment the checker sync counteruntil the current memory address is the same as the firstsynchronization point. In other words, the scheduler module 312 countsuntil the checker CPU 204 reaches the first synchronization point.

The scheduler module 312 selectively adjusts a synchronization densitybased on the value in the master sync counter and the value in thechecker sync counter. For example, a synchronization point can be addedat a task level or a time slice level based on whether the values storedin the master sync counter and the checker sync counter are greater thana density threshold.

In another implementation, the processor module 316 and the processormodule 320 each receive the plurality of tasks to execute from the firstand second copy of the application software. In order to ensure that theprocessor module 316 and the processor module 320 are executingidentical tasks in synchronization, prior to executing the task, theprocessor module 316 communicates with the processor module 320 via aninter-processor communication (IPC) channel. For example only, the IPCchannel may be dual-redundant, high speed, point-to-point serialcommunication link between the master CPU 204 and the checker CPU 208.

Each of the processor modules 316 and 320 generate a plurality ofsynchronization messages based on operating system events. The operatingsystem events may include a task requesting a time instruction, arequest sending and/or receiving a data packet, or any other suitableoperating system event. The processor module 316 generates a firstsynchronization message when the master CPU 204 reaches thesynchronization point. The processor module 316 communicates the firstsynchronization message to the processor module 320. Similarly, theprocessor module 320 generates a second synchronization message when thechecker CPU 208 reaches the synchronization point. The processor module320 communicates the second synchronization message to the processormodule 316.

The processor module 316 determines whether the processor module 320 hascommunicated a synchronization message. When the processor module 316determines the processor module 320 has not communicated asynchronization message, the processor module 316 continues to wait toreceive a synchronization message from the processor module 320. Theprocessor module 316 increments the synchronization counter. Conversely,when the processor module 316 determines the processor module 320 hascommunicated a synchronization message, the processor module 316determines whether the synchronization message indicates the processormodule 320 is synchronized with the processor module 316.

For example, the processor module 316 generates the firstsynchronization message based on a first task requesting a first timeinstruction. The processor module 320 generates the secondsynchronization message based on a second task requesting a second timeinstruction. The processor module 316 receives the secondsynchronization message from the processor module 320. The processormodule 316 determines whether the master CPU 204 and the checker CPU 208are synchronized based on the first and second synchronization messages.

For example, when the processor module 316 compares the firstsynchronization message to the second synchronization message. When theprocessor module 316 determines the first synchronization message is thesame as the second synchronization message, the processor module 316determines the master CPU 204 and the checker CPU 208 are synchronized.When the processor module 316 determines the master CPU 204 and thechecker CPU 208 are synchronized, the processor module 316 generates afirst processor synchronization signal to the processor module 320.

Similarly, the processor module 320 determines whether the processormodule 316 has communicated a synchronization message. When theprocessor module 320 determines the processor module 316 has notcommunicated a synchronization message, the processor module 320continues to wait to receive a synchronization message from theprocessor module 316. The processor module 320 increments thesynchronization counter. Conversely, when the processor module 320determines the processor module 316 has communicated a synchronizationmessage, the processor module 320 determines whether the synchronizationmessage indicates the processor module 316 is synchronized with theprocessor module 320.

For example, the processor module 316 generates the firstsynchronization message based on a first task requesting a first timeinstruction. The processor module 320 generates the secondsynchronization message based on a second task requesting a second timeinstruction. The processor module 320 receives the first synchronizationmessage from the processor module 316. The processor module 320determines whether the master CPU 204 and the checker CPU 208 aresynchronized based on the first and second synchronization messages.

For example, when the processor module 320 compares the firstsynchronization message to the second synchronization message. When theprocessor module 320 determines the first synchronization message is thesame as the second synchronization message, the processor module 320determines the master CPU 204 and the checker CPU 208 are synchronized.When the processor module 320 determines the master CPU 204 and thechecker CPU 208 are synchronized, the processor module 320 generates asecond processor synchronization signal to the processor module 316.

The processor module 316 and the processor module 320 determine whetherto execute tasks associated with the first and second copy of theapplication software based on the first processor synchronization signaland the second processor synchronization signal. For example, theprocessor modules 316 execute tasks associated with the first and secondcopies of the application when the processor modules 316 generates thefirst processor synchronization signal and receives the second processorsynchronization signal. Similarly, the processor modules 320 executetasks associated with the first and second copies of the applicationwhen the processor modules 320 generates the second processorsynchronization signal and receives the first processor synchronizationsignal.

Further, the processor module 316 and the processor module 320 determinewhether to send and/or receive an Ethernet packet based on the first andsecond processor synchronization signals. For example, the processormodules 316 and 320 send and receive data packets associated with thesystem 300. The data packets may include the plurality of data packetsdescribed with respect to FIGS. 1 and 2. By way of non-limiting exampleonly, the processor module 316 may generate a first outgoing datapacket. The processor module 316 transmits the first outgoing datapacket when the processor module 316 generates the first processorsynchronization signal and receives the second processor synchronizationsignal.

Similarly, the processor module 320 may generate a second outgoing datapacket. The processor module 320 transmits the second outgoing datapacket when the processor module 320 generates the secondsynchronization signal and receives the first processor synchronizationsignal. It is understood that while only transmitting data packets isdescribed, the principles of the present disclosure also apply to eachof the processor modules 316 and 320 receive data packets.

In yet other implementations, the granularity of synchronization may becontrolled based on a density of the synchronization point. For example,if the operating system is sending multiple data packets, it can use asingle checkpoint to communicate the number and/or sequence of packetsto be sent, thus reducing the number of checkpoints required to maintainsynchronization.

The scheduler module 312 determines whether to restart the system 300based on the synchronization counter. For example, the schedule module312 compares a value of the synchronization counter to a predeterminedthreshold. The predetermined threshold may be a value associated with anumber of missed synchronization points. For example, the predeterminedthreshold may be 3. In other words, once the scheduler module 312determines that the master CPU 204 and the checker CPU 208 were out ofsynchronization 3 times. When the scheduler module 312 determines thevalue in the synchronization counter is greater than the predeterminedthreshold, the scheduler module 312 instructs the system 300 to restart.

With reference to FIG. 4, a flow diagram illustrating a processorsynchronization method 400 according to the principles of the presentdisclosure starts at 404. At 408, the method 400 resets the masterregister B and the checker register B. At 412, the method 400 writes thecurrent master CPU address to the master register A. At 416, the method400 writes the current checker CPU address to the checker register A. At420, the method 400 writes the result of a logical AND of the masterregister A and the checker register A to the master register B. At 424,the method 400 writes the result of a logical AND of the master registerA and the checker register A to the checker register B. At 428, themethod 400 determines whether the value in the master register B is thesame as the value in the checker register B. If true, the methodcontinues at 432. If false, the method continues at 412. At 432, themethod 400 generates a processor synchronization signal. At 436, themethod 400 releases the master CPU and the checker CPU. The method endsat 440.

With reference to FIG. 5, a flow diagram illustrating an alternativeprocessor synchronization method 500 according to the principles of thepresent disclosure starts at 504. At 508, the method 500 generates afirst synchronization message. At 512, the method 500 determines whethera second synchronization message was received. If true, the method 500continues at 520. If false, the method 500 continues at 516. At 516, themethod 500 increments a synchronization counter. The method 500continues at 512. At 520, the method 500 determines whether the firstsynchronization message is the same as the second synchronizationmessage.

If true, the method 500 continues at 528. If false, the method 500continues at 524. At 524, the method 500 increments the synchronizationcounter. The method 500 continues at 512. At 528, the method 500generates a first synchronization signal. At 532, the method 500determines whether a second synchronization signal was received. Iftrue, the method 500 continues at 536. If false, the method 500continues at 532. At 536, the method 500 releases the master CPU 204 andthe checker CPU 208. The method 500 ends at 540.

The foregoing description of the embodiments has been provided forpurposes of illustration and description. It is not intended to beexhaustive or to limit the disclosure. Individual elements or featuresof a particular embodiment are generally not limited to that particularembodiment, but, where applicable, are interchangeable and can be usedin a selected embodiment, even if not specifically shown or described.The same may also be varied in many ways. Such variations are not to beregarded as a departure from the disclosure, and all such modificationsare intended to be included within the scope of the disclosure.

As used herein, the term module may refer to, be part of, or include anApplication Specific Integrated Circuit (ASIC); an electronic circuit; acombinational logic circuit; a field programmable gate array (FPGA); aprocessor (shared, dedicated, or group) that executes code; othersuitable hardware components that provide the described functionality;or a combination of some or all of the above, such as in asystem-on-chip. The term module may include memory (shared, dedicated,or group) that stores code executed by the processor.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes,and/or objects. The term shared, as used above, means that some or allcode from multiple modules may be executed using a single (shared)processor. In addition, some or all code from multiple modules may bestored by a single (shared) memory. The term group, as used above, meansthat some or all code from a single module may be executed using a groupof processors. In addition, some or all code from a single module may bestored using a group of memories.

The apparatuses and methods described herein may be implemented by oneor more computer programs executed by one or more processors. Thecomputer programs include processor-executable instructions that arestored on a non-transitory tangible computer readable medium. Thecomputer programs may also include stored data. Non-limiting examples ofthe non-transitory tangible computer readable medium are nonvolatilememory, magnetic storage, and optical storage.

What is claimed is:
 1. A method for synchronizing central processingunits (CPU) comprising: generating, by a first processor, a firstsynchronization message when a first CPU reaches a synchronization pointwithin a first series of instructions; generating, by a secondprocessor, a second synchronization message when a second CPU reachesthe synchronization point within a second series of instructions,wherein the first and second series of instructions are identical;communicating, by the first processor, the first synchronization messageto the second processor; communicating, by the second processor, thesecond synchronization message to the first processor; determining, bythe first processor, whether the second synchronization message wascommunicated; determining, by the second processor, whether the firstsynchronization message was communicated; and determining, by one of thefirst processor or the second processor, whether to execute a pluralityof identical tasks in the respective first and second series ofinstructions based on whether the respective second or the firstsynchronization messages have been received by the one of the firstprocessor or the second processor and whether the first and secondsynchronization messages are identical.
 2. The method of claim 1 furthercomprising incrementing a synchronization counter in response to notreceiving at least one of the first and second synchronization messages.3. The method of claim 2 further comprising selectively determiningwhether to restart a first CPU and a second CPU in response to thesynchronization counter being greater than a predetermined threshold. 4.The method of claim 1 further comprising determining whether the firstCPU and the second CPU are synchronized based on a comparison of thefirst and second synchronization messages.
 5. The method of claim 4further comprising determining whether the first CPU and the second CPUare synchronized when the first synchronization message is generated andthe second synchronization message is received.
 6. The method of claim 5further comprising generating a first processor synchronization signalin response to the determination that the first CPU and the second CPUare synchronized.
 7. The method of claim 4 further comprisingdetermining whether the first CPU and the second CPU are synchronizedwhen the second synchronization message is generated and the firstsynchronization message is received.
 8. The method of claim 7 furthercomprising generating a second processor synchronization signal inresponse to the determination that the first CPU and the second CPU aresynchronized.
 9. The method of claim 8 further comprising executing theplurality of tasks in response to receiving the first processorsynchronization signal and the second synchronization signal.
 10. Themethod of claim 8 further comprising not executing the plurality oftasks in response to not receiving at least one of the first and secondsynchronization signals.
 11. A method for synchronizing centralprocessing units (CPU) comprising: generating, by a first processor, afirst synchronization message when a first CPU reaches a synchronizationpoint within a first series of instructions; generating, by a secondprocessor, a second synchronization message when a second CPU reachesthe synchronization point within a second series of instructions,wherein the first and second series of instructions are identical;communicating, by the first processor, the first synchronization messageto the second processor; communicating, by the second processor, thesecond synchronization message to the first processor; determining, bythe first processor, whether the second synchronization message wascommunicated; determining, by the second processor, whether the firstsynchronization message was communicated; and determining, by one of thefirst processor or the second processor, whether to execute at the sametime a plurality of identical tasks in the respective first and secondseries of instructions based on at least one of: whether the respectivesecond or the first synchronization messages have been received by theone of the first processor or the second processor, or a comparison ofthe first and second synchronization messages and whether the first andsecond synchronization messages are identical.
 12. The method of claim11 further comprising incrementing a synchronization counter in responseto not receiving at least one of the first and second synchronizationmessages.
 13. The method of claim 12 further comprising selectivelydetermining whether to restart a first CPU and a second CPU in responseto the synchronization counter being greater than a predeterminedthreshold.
 14. The method of claim 11 further comprising determiningwhether the first CPU and the second CPU are synchronized based on thecomparison of the first and second synchronization messages.
 15. Themethod of claim 14 further comprising determining whether the first CPUand the second CPU are synchronized when the first synchronizationmessage is generated and the second synchronization message is received.16. The method of claim 15 further comprising generating a firstprocessor synchronization signal in response to the determination thatthe first CPU and the second CPU are synchronized.
 17. The method ofclaim 14 further comprising determining whether the first CPU and thesecond CPU are synchronized when the second synchronization message isgenerated and the first synchronization message is received.
 18. Themethod of claim 17 further comprising generating a second processorsynchronization signal in response to the determination that the firstCPU and the second CPU are synchronized.
 19. The method of claim 18further comprising executing the plurality of tasks in response toreceiving the first processor synchronization signal and the secondsynchronization signal.
 20. The method of claim 18 further comprisingnot executing the plurality of tasks in response to not receiving atleast one of the first and second synchronization signals.
 21. A methodfor synchronizing central processing units (CPU) comprising: generatinga first memory address to a first memory register when a first CPUreaches a synchronization point; generating a second memory address to asecond memory register when a second CPU reaches the synchronizationpoint; comparing, in the first CPU, the first memory register to thesecond memory register and storing a result of the comparison in a thirdmemory register; comparing, in the second CPU, the first memory registerto the second memory register and storing a result of the comparison ina fourth memory register; comparing the third and fourth memoryregisters; and determining whether to execute at the same time aplurality of identical tasks based on the result of the comparisonbetween the third and fourth memory registers.
 22. The method of claim21 further comprising determining whether the first CPU and the secondCPU are synchronized based on the comparison between the third andfourth memory registers.
 23. The method of claim 22 further comprisingdetermining the first CPU and the second CPU are synchronized when thethird and fourth memory registers are identical.
 24. The method of claim23 further comprising generating a first processor synchronizationsignal in response to determining that the first CPU and the second CPUare synchronized.
 25. The method of claim 24 further comprisinggenerating a second processor synchronization signal in response todetermining that the first CPU and the second CPU are synchronized.